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ABSTRACT 


Computer software has taken an increasingly larger role in the U.S. Navy. It is 
used in nearly every facet of naval operations, from administrative chores to controlling 
complex weapons systems. Because of the high cost of software and the potential for 
inadvertent misuse, it is important that software be easy to use and understand. This 
thesis explores the methods and techniques available for Sssndicaae software usability 
evaluations. Using what described in this thesis, actual software usability testing is done 
on a recently developed Web site. The Web site [http://spitfire.avsafety.nps.navy.mil] 
evaluated in this study is designed to allow aviation units to complete a safety survey 
online. This thesis describes the usability test conducted on the Aviation Command 
Safety Assessment (ACSA) Web site and establishes a methodology that can be used on 
any future Navy Web site. The results of this usability test show that improvement can be 


made to the interface design and presentation of material. 
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I. INTRODUCTION 


A. PURPOSE 


The purpose of this thesis is to develop and apply a methodology that can be used 
to assess the usability of a Web-based user Interface for administering the Aviation 
Command Safety Assessment (ACSA) Survey questionnaire via Internet Technology. 
This questionnaire is needed to support the School of Aviation Safety in applying a 
methodology for assessing the effectiveness of Naval organizations in the management of 
risks associated with flight operations. 

In April 1996, Commander Naval Air Pacific chartered an Aviation Safety Quality 
Management Board (QMB) to explore ways in which the Naval aviation mishap rate 
might be lowered. In order to establish a baseline analysis of the “safety climate” at the 
squadron level, Dr. Anthony Ciavarelli was appointed as the principal investigator for the 
Aviation Safety Command Survey. He developed a command safety assessment 
questionnaire that was distributed to 69 squadrons in the fall of 1996. The results of this 
initial survey provided valuable insight into the safety climate of squadrons, wings, and 
Type Commanders and Naval aviation as a whole. The process for completing this 
survey was a paper questionnaire, with submissions being returned on a form that could 
be read by a scanner. Distributing, administering, collecting and then analyzing these 


surveys was an expensive, time consuming and labor-intensive effort. 


In March 1999, an internet-based, prototype mode] for administering this survey 


using new technologies that allowed automated survey submission and analysis was 














by two Naval Postgraduate students (Held and Mingo, 1999). The results of their thesis 
is a web site [http://spitefire.avsafety.nps.navy.mil] that adheres to a three-tier 


client/server architecture. 


Experience has shown that the value of computer systems, including web sites is 
directly tied to their usability. Therefore, this thesis will develop guidelines for building 
Web sites which focus on building a human-computer-interface that satisfies ease of use 


standards. 


B. OBJECTIVE 


In order to achieve the stated purpose of this thesis three specific objectives were 
accomplished. First, a selected review of the related literature on Web site design and 
usability evaluation was conducted. This literature review yielded an understanding of 
the most effective and insightful methods for usability engineering for the web. Second, 
using the knowledge from the literature review, a methodology was developed for 
usability testing and applied to the Aviation Command Safety Assessment (ACSA) 
Survey Web site. In addition to the formal investigation of usability, input from aviation 
safety officers who used the web survey was solicited. Finally, the results of the usability 
evaluation were analyzed and summarized in order to capture the benefits of this study for 


future web development. 


C. PROBLEM STATEMENT 


As previously indicated, Held and Mingo spent at least a year and several 





thousand dollars in development and implementation of the Aviation Command Safety 











Assessment (ACSA) Survey Web site. This effort does not guarantee that a successful 
Web site for completing the survey has been produced. No matter how good the Web site 
looks, if the intended user has difficulty using the Web site, then the value is minimized. 
Over time, common flaws associated with poor human-computer interface (HCI) design 
have been identified. Some of the most prevalent problems involve screen presentation 
formats and navigation between various screens (Gillespie, 1998). Poor design also 
results from a failure, on the part of the HCI, to reflect the form and level of user 
guidance considered most appropriate. If the HCI is poorly designed, then the potential 
exists for the ultimate user of the Web site to reject it simply because it is too difficult to 
use. To avoid this possibility it is important to determine what, if any, ease of use 
problems may exist in the Aviation Command Safety Assessment (ACSA) Survey Web 
site interface design. This determination relies on ascertaining and applying a method of 
usability evaluation that answer such questions as: 

e Are the instructions easy to read and understand? 

e Does the arrangement of information on the screen make sense? 

e Are users able to quickly learn how to complete the survey? 

e Is navigation within the Web site easy to accomplish? 

e Are the selection of colors, icons and graphics intuitive? 

By investigating these and other questions, this study provides a preliminary 
indication of the success or failure of the Aviation Command Safety Assessment (ACSA) 


Survey Web site and potential improvements that may be made to the program. 








D. SCOPE AND LIMITATIONS 


It is not the intent of this research to provide an in-depth study of the Aviation 
Command Safety Assessment (ACSA) Survey questionnaire, or the actual development 
of the Aviation Command Safety Assessment (ACSA) Web site. Prior studies, 
specifically Ciavarelli and Figlock, et. al, (1996), contain substantial coverage of these 
subjects and web site development has been completed by Held and Mingo (1999). 
Furthermore, the limitations and computer hardware requirements needed to administer 
and access the web site are well documented in these publications and will not be 
discussed here. The focus of this study is the Aviation Command Safety Assessment 
(ACSA) Web site design, or more specifically, the interface provided by the software as it 


is run on a typical personal computer. 


E. THESIS ORGANIZATION 


This thesis consists of five chapters. Chapter II provides a review of selected 
literature pertinent to web site design, software usability and usability testing. Chapter II 
covers the methodology behind the data collection and Chapter IV, the analysis of the 
data collected. The remaining chapter, Chapter V, provides the conclusion and 
recommendations. The questionnaire, interview transcripts and other items significant to 


this study are attached as appendices. 











II. BACKGROUND 


Building a medium or large World Wide Web site, whether for distribution over 
the Internet or over an Intranet, can and should be viewed as a major software 
development effort. Applications on the Web range from having users complete simple 
tasks, such as filling in a form, to more complex undertakings, such as collaborating on a 


project. 


The tools available for creating Web pages allow anyone to design a Web 
application. But the Web--unlike Windows, Macintosh, and Unix--has no standards for 
designing applications. People come to the Web with expectations based on using the 
other platforms, and these expectations make up their conceptual model of using an 


application on the Web. 


Frequently, in Web applications users get lost because the model of navigation on 
the Web differs from that in other applications. This difference causes confusion among 
users, but application developers can use a few techniques to help people successfully use 


the application. 


Once Web site creation is seen as software development, it becomes natural to 
apply the tools and methods we have learned in past projects. The life cycle of Web 
creation is identical to that of traditional software: requirements gathering, analysis, 
design, implementation, testing, and deployment. And, just as traditional software 
development should have a functionality and a usability component, so should Web 


development efforts. 











A. WHAT IS USABILITY? 


























Usability can be defined as the degree to which a given piece of software assists 
the person sitting at the keyboard to accomplish a task, as opposed to becoming an 
additional impediment to such accomplishment. The broad goal of usable systems is 
often assessed using several criteria: 

e Ease of learning. 

e Retention of learning over time. 

e Speed of task completion. 

e Error rate. 

e Subjective user satisfaction. 

Methodologies for building usable systems have been introduced and refined over 
the past fifteen or so years under the discipline of Human-Computer Interaction (HCD. 
HCI principles include an early and consistent focus on end users and their tasks, 
empirical measurements of system usage, and iterative development. Much effort has 
been put into exploring cognitive models of human behavior as it relates to computer 
usage, and developing guidelines for screen layout and system dialogues. These are 
predictive endeavors whose purpose is to assist the software developer in the initial task 
analysis and system design. 

But, just as comprehensive functional requirements and a detailed design 


document do not by themselves guarantee that a programmer’s final code will be correct, 





so up-front usability guidelines do not by themselves guarantee a usable end product. In 


both cases a distinct validation process is required. 





Usability engineering techniques can create more usable Web sites by helping 
developers focus on the users rather than the technology. Understanding users’ needs and 
the tasks they are trying to accomplish is steadily becoming more important than adopting 
the latest whiz-bang feature. Of course, if the latest technology fulfills a need, it should be 
used; but incorporating this technology just to be cool is becoming less and less 
acceptable. | 

Improving usability is not a black art: it can be accomplished by heeding common 
sense advice and applying techniques that help us understand and meet the needs of users. 
A good introduction to this topic is Jakob Nielsen’s Usability Engineering (Nielsen 1994). 
Applying usability engineering to the Web is not that difficult, although the Web does 
introduce a few interesting problems--and solutions. Darrell Sano’s Designing Large- 
Scale Web Sites (Sano 1996) discusses many Web usability issues, such as limitations 
and constraints for Web publishing and structuring the information space. 

What follows is a review of usability techniques and methods, seen in light of the 
constraints and tradeoffs of the World Wide Web. The best place to see usability 
engineering applied to the Web is the Sun-on-Net project (Nielsen 1997c), where 
usability testing, iterative design, and other activities improve the usability of Sun’s Web 


Site. 


B. INTERFACE DESIGN CONSIDERATIONS 
1. Reading on the World Wide Web 


In order to design interfaces geared toward Web based clients one must first study 











how users read from that medium. According to the Nielsen Norman Group, “User 
experience” encompasses all aspects of the end-user’s interaction with the organization, 


its services, and its products. 

The first requirement for an exemplary user experience is to meet 
the exact needs of the customer, without fuss or bother. Next comes 
simplicity and elegance that produce products that are a joy to own, a joy 
to use. True user experience goes far beyond giving customers what they 
say they want, or providing checklist features. In order to achieve high- 
quality user experience in a company’s offerings there must be a seamless 
merging of the services of multiple disciplines, including engineering, 
marketing, graphical and industrial design, and interface design. (Nielsen 


Norman Group http://www.nngroup.com/about/ 

userexperience.html, February 22, 1999). 

These statements can easily be applied to any government scenario, where the customer is 
analogous to any user of an organization’s Web site and references to the company equate 
to the organization using a Web site for constructive purposes. 

A study conducted by John Morkes and Jakob Nielsen found that 79 percent of 
test users scanned and new page they came across, with only 16 percent reading word-for- 
word (Nielsen, 1997a). Four possible reasons for this particular behavior are (Nielsen, 
1997b): 


e Reading from computer screens is tiring and approximately 25 percent 
slower than reading from paper; 


e The Web is a user-driven medium, i.e. users feel they must move on 
and click on things; 


e Each page has to compete with literally millions of other pages for the 
user’s attention; 


e Modern life is hectic. 
According to research by Nielsen and others, individuals do not want to invest too much 


time in any one page while searching for information on the Web because the page may 














not contain the needed information. Thus, the page is quickly scanned for relevance in a 
quick once over before clicking on the next page. 

Other important issues are the credibility of the information, since there is no 
guarantee about who has provided the information posted to the Web site. Studies by 
Nielsen have shown that by providing hypertext links to external related sites, using high 
quality graphics, and employment of good writing practices, credibility or perceived 
credibility of the information on a site can be significantly increased. 

2. Writing for the World Wide Web 

By understanding how users interface with and read on the Web, insight can be 
grained into methods of effective writing for the Web. Information should be presented 
in a concise and intuitive format (Gahran, 1998a). Keep pieces short, by breaking up if 
plausible and following these guidelines (Gahran, 1998b ): 

e Pare down whatever you are saying to the greatest extent that is 

appropriate. Consider using bulleted lists and highlighted key 


words. 


e Make sure every bit of content has a point that will matter to 
the user, and make those points obvious. 


e Break up longer stretches of text into manageable “chunks”. 
e Attempt to predict and answer all the major natural questions 
that your users will probably have about your organization or 
your site’s content. Consider dropping any content that does 
not focus on what the user probably wants to know. 
Writing for the Web is more akin to newspaper journalism than to any other form 
of distributing information, including marketing publications, magazine articles, books or 


theses. In the newspaper format, the summary or conclusions of the story are at the front 


of the story. This is done in order to immediately capture and retain the reader’s attention. 











If the reader these lines and is interest in the topic, they will read the entire article and be 
able to retrieve details. This is done in order to immediately capture and retain the 
reader’s attention. If the reader reads these lines and is interested in the topic, they will 
read the entire article to retrieve details. This is especially true for the front page in the 
area known as “above the fold.” This prime real estate must be used to full advantage to 
make the sale by hooking the potential customer with interesting facts in order to entice 
them to purchase the paper. In the same vein, each Web page on a site must make promise 
of the information contained therein, as well as in the lower “drill-down” layers. The most 
sought after information should be summarized or encapsulated at the top of the page, in 
the writing style described above. The user will scan this page and determine if it contains 
the information needed, and read further or drill-down for more details. Nielsen (1996a) 
refers to this type of writing as the inverted pyramid style. This allows the page creator to 
identify the purpose of the site near the top of the page so the reader can easily see if they 
want to invest time at that site. 

Creating hyperlinks in text serves to highlight that text and to provide a place so 
the user can dill-down to obtain more information, as desired. This is one of the most 
powerful features of the Web, and this is what creates the “webbing” effect of the Internet. 
This allows Web writers to split their writing into smaller, coherent pieces to avoid long 


scrolling pages (Nielsen, 1996b). 


C. WEB USABILITY ENGINEERING LIFE CYCLE 


The usability engineering life cycle (see the following list Nielsen 1994) 


represents a menu of choices that can be worked into the broader development context in 











order to increase usability. The techniques focus on understanding users, promoting 
usable designs, finding usability problems, and understanding other constraints in 
development. 

1. Know the user. 

2. Competitive analysis. 

3. Setting usability goals. 

4. Parallel design. 

5. Participatory design. 

6. Coordinated design of the total interface. 

7. Applying guidelines and heuristic analysis. 

8. Prototyping. 

9. Empirical testing. 

10. Iterative design. 

11. Collecting feedback from field use. 
Steps 1,7, 10 and 11 (Know the user, Applying guidelines and heuristic analysis, Iterative 
design, and Collecting feedback from field use) are most pertinent to Web usability and 


are elaborated upon in the sections that follow. 


1. Know the User 

The most basic of usability guidelines, "know the user," can be very difficult on 
the Web, where international access, exponential growth, and wide-ranging demographics 
(from school-age children to elderly citizens) are the norm. It is also difficult to understand 


what people really want from this new medium, in which so much is possible. The many 
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surveys of general Internet demographics will provide some information on "who is out 
there," but talking to users is still the best way to get a handle on user-specific usability 


problems. 


On the Web, "know the user" also means knowing the speed of their Internet 
connection, which browser version they are using, what plug-ins they have, and so on. 
Understanding these factors plays a big part in choosing advanced Web technologies and 


determining when they can be safely introduced. 


2. Applying Guidelines and Heuristic Analysis 

Many usability problems can be avoided by following the published guidelines 
(Nielsen, 1996c) and by adhering to some general rules of thumb (see the Heuristics 
section below). Since the guidelines and heuristics are very general, additional usability 


techniques, such as empirical testing, are needed to determine the best solution for each 


particular problem. 


3. Iterative Design 
Iterative design involves a four-step process: 


e Creating something. 
e Testing it out with users. 
e Understand the strengths and weaknesses. 
e Designing a better version. 
This fits very well into the Web "culture," in which a site does not have to be 


perfect the first time and the technology is constantly changing. Iterative design works 


best if it is part of the overall Web management process rather than an extra chore 
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performed every year or so. Between iterations, one should be gathering user feedback, 
analyzing log data, creating rapid prototypes, and testing users. 

Part of the iterative design should also be an evaluation of new technologies for 
the next version. Such a technology should not drive the change but rather be seen as a 


solution to known usability problems. 


4. Collecting Feedback from Field Use 

One of the easiest and most effective ways to improve the usability of a Web site 
is to see what real users do with it. A strength of the Internet is the ease with which the 
users can provide feedback. A common practice is to place mail to and feedback buttons 
on every page, encouraging users to take a moment to send the developers a note if they 
have a problem, a complaint, or some praise. The best user feedback can be designed into 
the site transparently. 

Good logging of Web site usage is important for those who have to justify the cost 
of developing and supporting a site. As far as improving usability, logs can also indicate 
how people access the site, what problems they are having, and how a redesign affects 


usage. 


D. HEURISTICS 


Heuristic evaluation is one of the most important aspects of usability sageedigs 
it is easy, fast, and inexpensive. It involves study of a user interface by a small set of 
evaluators for violations of common usability principles (rules of thumb, heuristics) so 
that the next iterative design can try to solve the usability problems. See the list that 


follows for some of the common usability heuristics (Nielsen 1994). 
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e Simple and natural dialog. 

e Speak the users’ language. 

e Minimize the users’ memory load. 
e Consistency. 

e Feedback. 

e Clearly marked exits. 

e Shortcuts. 

e Good error messages. 

e Prevent errors. 


e Help and documentation. 


Heuristic evaluation is well suited to the Web, where everyone is in a hurry and 
the site can be evaluated from half-way around the world. When doing a heuristic 
evaluation for the Web, the common heuristics of consistency, feedback, and so on apply. 


But a few other rules of thumb are also useful when evaluating a Web user interface. 


1. User Control 

Because of the Web’s architecture, in which browsers are responsible for 
interpreting HyperText Markup Language (HTML), developers never really know who-- 
or what--will be processing their pages. Forcing users into particular fonts, sizes, colors, 
screen widths, or browser versions will generally reduce usability. Of course, designers 
need to have some control. But the more general the implementations and designs, the 
more likely we will reach satisfactory levels of usability for the widest range of users and 


over the longest periods of time. 
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Emerging Web technologies should often be avoided because they fail on this user 
control heuristic early in their life spans. One example is animation. To accommodate a 
diverse set of situations, users should be allowed to start and stop animation on their own. 
On one hand, animations that repeat forever are distracting; on the other, those that 
cannot be repeated lose their value. Until users gain adequate control over this technology 
on the Web, animations should be used very carefully. (Even once user control is in 
place, animation should be used only when appropriate (Nielsen 1995). 

A new Web feature that will enable greater user control is cascading style sheets 
(Andrews 1996). Style sheets force the developer to separate the presentation aspects of 
the page from the content, making it easier for a user to "intercept" the presentation 
information and insert his or her own requirements for display. Beyond selecting link 
color, for example, style sheets will extend the amount of control users have to include 
many other items of personal taste. The largest gain, however, will be that users can 
specify style attributes in a standard way to ensure their own personal requirements for 
usable access. For example, users with poor eyesight could specify large font sizes for 
headers and text, overriding the font sizes specified by the authors. 

When evaluating a site for user control, the following questions can be asked to 
determine what usability problem might arise: 

e Can users override this feature? 

e Can they customize to suit their tastes or needs? 


e Will giving users control of this feature reduce the usability of the site? 
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2. Structure 

In all hypermedia systems, some underlying structure is important to help users 
figure out where they are and where they can go next. On the Web, where search engines 
and links from other places can throw users into the middle of any site, showing this 
structure is even more important. Since Web browsers lack substantial navigation aids to 
help users discover this structure, Web authors have to do a lot of this work themselves. 


To evaluate structure, one can look at every major page of a site and ask the following 


questions: 


e Ifauser were taken directly to this page from an outside site, what could they 
figure out about the rest of the site from this one page? 

e Is the site "brand" present? 

e Is it clear which part of the site they are in? 


e Js it clear how to navigate to other parts of the site? 


3. Design for Change 
The Web is about change, so a user interface had better be able to deal with both 


continually changing content and design. The evaluator can ask the following questions: 


e Howis old content archived? 
e How is new content added? 


e Can this design withstand the addition of 20 times the current content? 
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If everything has to be introduced through a "What’s new" page, the site was not 
properly designed for changing content. If iterative design is going to work well, the 
design has to be flexible enough to withstand small changes without having to be thrown 
away. For example, if user feedback indicates a need for an additional major section to a 
Web site, can the design handle it, or would the site have to be redone from scratch? If 


improving the usability is too difficult, it simply will not happen. 


E. USABILITY TRADEOFFS 

Usable interface design involves tradeoffs: weighing the costs and benefits to 
develop the best solution under the current conditions. The Web requires some very 
serious tradeoff decisions because of its low bandwidth, the role of browsers, and the 
existence of standards. 

It is important to note that these tradeoffs are not meant to mandate one decision 
over another. What is important is to understand the pros and cons in order to make an 
informed decision. Sometimes usability has to be sacrificed because of cost, time, or 
management issues, but blindly adopting a new Web technology rather than studying the 


usability tradeoffs is not appropriate. 


For example, third generation site design (Lie 1996) improves usability by 
providing better page layout: paragraphs are indented, headers are close to their 
associated paragraphs, and there are left margins. One of the costs, however, is that access 
with graphic loading turned off (to speed the process) presents a significantly less usable 


site; the precisely aligned text is no more, and "unloaded graphics" icons litter the screen. 
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So if one is willing to eliminate use by one segment of the Internet population, a third- 
generation site might be appropriate. (Note also that several other populations of users are 
disenfranchised by third-generation site design such as those who have disabilities 
Paciello 1997). Part of the tradeoff is found in the answers to the following questions: is 
this a very usable site for a few users, a fairly usable site for the majority of users, or a 


"good enough" site for all users? 


1. Bandwidth 

Access speed is currently one of the major constraints on design. The best design, 
no matter how good, will not be seen if it takes too long to download. Until access speed 
is no longer a major problem, tradeoffs involving the quality of the graphics, the 
complexity of the design, and the use of higher bandwidth advanced technologies will Be 
important to understand in creating usable Web sites. In addition, developers would 


benefit from observing users at very slow speeds. 


Web site developers must be cognizant of download times and design 
accordingly. According to a newspaper article for the Seattle Times Web site (Williams, 
1998) here is a wish list for Web users to Web designers: 


e Create pages that appear on the screen quickly. Do not load them up with big 
graphics. 


e Do not start sound files running without asking if it is wanted first. 
e No blinking, animated graphics, or dancing objects. 


e Make navigation easy. Keep the buttons in the same place as you move 
through the site. Those buttons and icons should be logical and labeled. 
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Graphics and images are usually the largest part of any Web page, and as such they tend 
to use up the most bandwidth. Omitting some or all graphics from a page is the quickest 
and easiest way to decrease download time of a Web page, although some smaller, well 
chosen graphics may make the page more readable. Graphics to be placed on a Web page 
should be chosen carefully. They must add value to the information and the page is being 
used to convey, not just be placed there for decorative purposes. Consider that if a page 
contained ten 100KB images, the download amount would be over one megabyte of 
information. This certainly does not bode well for using a modem connection. 

Another tip for designers to consider is the use of alternative or the “AL=” 
attribute. This allows the user to have a description of an object while it is loading into 
the browser. If the transfer is interrupted or “times out,” the user will know if the item is 
something they are interested in trying harder to obtain, either by reloading the page or 
returning later to download it again. This also describes objects to those who have image 
loading turned off in their browsers, or those who have browsers that do not support 
images. Values set in this attribute should be a sensible description of the object they 
represent. There are four types of images contained on Web pages; page toys, navigation 
icons, supplemental or interesting, and those that are critical for understanding the page. 


(Flavell, 1998) 


2. Standards 
Standards make the World Wide Web interoperable. The most important standard 
for developers is HyperText Markup Language (HTML). The latest version of HTML is 


4.0 (Raggett and Hertzbert 1996). HTML 4.0 has brought the major Web vendors 
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together, making the Web more interoperable and slowing the "tag wars" that were 
impeding the creation of usable Web sites across the different platforms. Knowing the 
HTML standard and following it when appropriate are crucial in making informed design 
and implementation decisions. 

In general, the more meticulous we are in following the standards, the more usable 
a site is for the largest audience. Any variation from the standard, even if it follows 


"Netscape’s standard," will have an effect on some part of the user population. It would 


help to consider the following questions: 


e Js that new tag useful enough for those who can take advantage of it? 


e Does it significantly affect those who cannot take advantage of it? 


The best extensions are those that give added functionality for the "haves" while 
having no negative effect on the "have nots." A good example of this is the background 
color attribute for table cells. Because they are not part of the older HTML 3.2 
specification, table cell background colors are ignored by those browsers that follow the 
standard and have not implemented this extension. Developers need to be aware that this 
is indeed not part of the standard and should ensure that their sites are still usable for 
those who do not have access to this added feature. 

Developers should also be aware of cases in which browsers are "forgiving"--i.e., 
those that create usable layouts of nonstandard HTML. For example, images are not 
allowed in PRE sections according to the HTML specification, but most browsers will 
generate reasonable interpretations of such markup. Designs that rely on such 


nonstandard use of HTML should be carefully considered. 
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Designers of a corporate Intranet have it much easier than those with an Internet 
audience. If all users are known to have a particular browser and version, one can more 
safely take advantage of nonstandard aspects. Still, any nonstandard usage could tie that 


corporation to one particular vendor, so some intelligent decisions need to be made. 


3. Browsers 

Another tradeoff occurs because of the crucial role that browsers play on the Web. 
The browser does certain things that are out of the author’s control--history, bookmarks, 
and exact page layout, for example. The fact that the developer has to rely on the browser 
to provide certain functionality may limit the creation of usable sites. One need only look 
at the sophisticated and easy-to-use interface on CD-ROMs today to realize the limits that 
browsers impose on the Web. Browsers often have bugs that make certain features 
unusable. Netscape Navigator’s early implementation of frames had serious problems 
with Back and bookmarking, which made frames quite unusable. (Now those bugs are 
fixed and frames are more usable--but not necessarily usable enough for certain 
applications.) Since not everyone has the most current version of browsers, these bugs are 
still an issue when designing a site today. Finally, there are a lot of browsers, each acting 
slightly different. Developers need to test their sites on as many platforms as possible in 


order to ensure usability for the widest range of visitors. 


F. USABILITY TESTING 


Solid design and usability are features explicitly used to sell products to 
consumers. When we hear someone selling Windows 95™ as being "just like a.Mac," or 


an Infiniti salesman preaching that thousands of customers were tested in order to 
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determine the placement of displays in Infiniti automobiles, this suggests that a payoff 
exists in terms of customer satisfaction and sales. 

There are methods available to web designers that may not have been applied 
previously to legacy system design. Bellcore’s User-Centered Design (UCD) methodology 
helps designers to rethink and redesign existing software systems to accommodate users 
and the web. UCD is a process for creating software that meets "documented objectives 
for usability and usefulness (with) an early and continuous focus on users throughout the 
analysis, design, and development process" (Salasoo, White, Dayton, Burkhart, and Root, 
1994). It is never too late to start usability testing, but the earlier one starts, the larger the 
payoff in time savings and user satisfaction. A good example of how usability testing was 
applied to web design is presented by Jakob Nielsen and Darrell Sano (1995) who wrote 
about their interface design testing of the new Sun Microsystems web site. 

A user-friendly web interface can have many advantages over an existing legacy 
interface. These advantages include: | 

e Reduce errors due to a redesigned. 

e Increased user acceptance of the system. 

e Increase flexibility for changes and improvement to the system. 

e Increase number of platforms supported with a consistent interface across 


platforms. 


G. CONCLUSION 


Usability engineering techniques, such as user testing, iterative design, heuristic 
evaluation, and user feedback, can make Web sites easier to use. Usability evaluations of 


new and emerging Web technologies, such as frames, JavaScript, Java and animated GIFs 
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(Graphics Interchange Format), are particularly important so that the tradeoffs of 
incorporating them into a site can be understood. Web standards, such as HTML 4.0, 
provide the basis of interoperability and help to ensure that designs will remain usable 
over time. Although excitement about the Web is still motivating users to play with new 
technologies, along with the increased use of the Web for “everyday” tasks and not just 
for fun, indicates a trend in which "what’s useful" will become more important than 
"what’s cool." Those sites that incorporate usability engineering into their development 
process will find that they are better able to survive in the highly competitive marketplace 


of the World Wide Web. 
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Il. METHODOLOGY 


A. SUBJECTS 

1. Background 

Selection of participants in this research focused on potential users of the Aviation 
Command Safety Assessment (ACSA) Web site and consisted of aviation units from 
Norfolk, Virginia and San Diego, California. This focus limited the research to 
Helicopter Combat Support Services squadrons HC-2, HC-3 and HC-11, Light Airborne 
Multi-Purpose System (LAMPS) squadron HSL-45 and Airborne Early Warning (AEW) 
squadrons VAW-120 and VAW123. These squadrons comprised a mix of rotary and 
fixed wing pilots. 

2 Participants 

From the available pilots in the squadrons, several were asked to participate in 
either the General Web Survey or usability testing of Aviation Command Safety 
Assessment (ACSA) Survey. All test subjects elected to participate voluntarily. No 
individuals with specific experience in the use of computers, software or the web site 
design were singled out for the purpose of including or excluding them from the 
evaluation. All participants received an overview of the Aviation Command Safety 
Assessment Web site goals and objectives and an explanation of their role in the usability 
testing. The participants were asked to be candid in their response to the General Web 
site, Aviation Command Safety Assessment (ACSA) Usability Survey, and interview as 


their responses would be confidential. 
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B. INSTRUMENTATION 

1. Questionnaire 

A General Web Survey was given to squadron volunteers in an attempt to assess 
the amount of integration of computer into the lives of aviation personnel. The intent of 
this survey was to profile the aviation user. By determining the level of familiarity of the 
target group with current computer technology, a baseline can be established by 
determining the least common denominator. Questions in this survey were designed to be 
as neutral as possible to determine the true user is perspective. The five basic 
characteristics of questions and answers that are fundamental to a good measurement 
process (Fowler, 1995) are as follows: 

® Questions need to be consistently understood. 


e Questions need to be consistently administered or communicated to 
respondents. 


e What constitutes an adequate answer should be consistently communicated. 


e Unless measuring knowledge is the goal of the question, all respondents 
should have access to the information needed to answer the question 
accurately. 


e Respondents must be willing to provide the answer called for in the question. 


How to design questions to measure subjective states (Fowler, 1995): 


Because there are no standards against which to evaluate the 
correctness or rightness of answers, standardization of the 
stimulus of the question is particularly critical in measuring 
subjective states. For this reason, designing questions that can 
be administered in a consistent way and that mean the same 
thing to all respondents, to the extent possible, is high on the list 
of strategies for creating good measurement of subjective states. 
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Equally important is standardizing the response task. That 
means clearly defining the dimension or continuum respondents 
are to use in their rating task and giving them a reasonable way 
to replace themselves, or whatever else they are rating, on that 
continuum. 

Fowler continues, the answers that come from a subjective questions have no 
absolute meaning, rather they are relative. The position of the answers relative to each 
other is where the relevant information is found. In general, surveys should ask things 
that respondents are able to reliably report. 

The first survey included general demographic questions about user characteristics 
to determine what the subjects had access too, and what they were accustom to using. 
There were also some questions based on a five point Lakers scale (Fowler 1995) 
regarding Web sites on the Internet itself, as well as questions relating to the time and 
amount of time spent on Web access. This survey was handed out to the participants and 
collected the next day. 

The Aviation Command Safety Assessment (ACSA) Usability Survey was 
designed to ascertain input about the ACSA Web site specifically. Questions were asked 
after the participants had on opportunity to explore the site. The first section of this 
particular survey was a set of general questions about the site based on a five point Likert 
scale. The second part was interactive, in which participants accessed a Web page and 
were asked questions about the pages, they directly observed. This portion was executed 
on a one-on-one basis, with the pages being accessed for the individual in an attempt to 


standardize the exercise. Participants were not asked to go to the site and find out 


answers to specific questions as in the study by Morkes and Nielsen (1997) on usability. 


2] 











This study concluded that by writing in concise, objective and scannable format, 
designers can greatly increase usability of a Web page. 

In order to assure anonymity, no names were taken and the data shall be displayed 
on in aggregate form. Appendices A and B show the General Web survey and the ACSA 
Usability Survey, respectively, while raw summary data are shown for each respective 
survey in Chapter IV. 

Zs Interviews 

All participants were interviewed upon completion of the Aviation Command 
Safety Assessment (ACSA) Usability Survey. As with the comments section of the 
questionnaire, the interviews were designed primarily to probe for frustration with 
specific parts of the ACSA. Only four questions were asked: 


1. Did you find the portion of the ACSA that you used to be an easy tool for 
completing the survey? 


Z. Was there anything that you found particularly difficult to do or 
understand? 


3. If you could change one thing about the ACSA Web site to make it easier 
to use, what would it be? 


4. Are there any features you would like to see added to the ACSA Web site? 


Significant responses to these questions are detailed in Appendix D. 


Cc. PROCEDURES 

A pretest walkthrough of log-on procedures was completed prior to the field 
evaluation. The walkthrough took place in several computer systems located at the Naval 
Aviation Safety School and reflected the usability test as planned for the actual testing of 


the ACSA Web site. Participants for this phase were staff volunteers representing typical 
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end users in the fleet. During the pretest walkthrough, questions to log-on procedures and 
interview questions were refined. 

The first usability test of the Aviation Command Safety Assessment was done at 
HC-11 San Diego, California. The volunteer participants were given a brief presentation 
on the ACSA Web site and an explanation of what was desired from them during the test 
(Appendix C). It was expressly noted to each participant that they were not the subject of 
the research but were only assisting in the evaluation of the ACSA Web site. After the 
presentation the subjects were scheduled to use the ACSA Web site so not to interfere 
with their other daily tasks. 

Prior to beginning each individual session the participants were familiarized with 
the log-on ‘pisces (Appendix B). The previous instructions were reiterated and the 
desire that the participants talk aloud during the session and point to the screen to indicate 
problems was stated. Participants were then instructed to proceed reviewing the Web site 
using the ACSA Usability Survey as a guide. Total time to complete the test was noted 


so that a basis for future tests could be established. 


D. ANALYSIS STRATEGY 

Descriptive statistics were used to examine the data from the questionnaire. The 
number and percentage of subjects who answered a question in a specific way is shown in 
a frequency table and illustrated with a bar graph. The number of participants in the test 
were used as a base for calculating the percentages. The mean, median and standard 
deviation were also calculated for the answers to each question. All tabulation and 


calculations were accomplished using Microsoft Excel spreadsheet software. 
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Significant comments, taken from the open ended portion of the questionnaires 


and interviews, are quoted in Appendices D and E respectively. 
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IV. RESULTS AND DISCUSSION 


A. DATA ANALYSIS 


The author conducted a survey of the aviation participants as described in the 
“survey Methodology” section of Chapter Ill of this thesis. The survey was conducted at 
each aviation squadron’s command as noted early. The survey was deemed to be self- 
explanatory, but the author was available to answer any questions. An early question 
concerned the meaning of intuitive. As found in Webster’s New World Dictionary, 
intuitive is defined as of, pertaining to, or arising from intuition; with intuition being the 
act or faculty of knowing without the use of use of rational processes; immediate 
cognition. Participants returned fourteen of the fourteen surveys given out. 

The findings presented in this section have been taken from the General Web 
Survey and Aviation Command Safety Assessment (ACSA) Usability Survey and 
interviews with the participants. 

The responses to the questionnaires were transferred to a Microsoft Excel 
spreadsheet (Appendices E and F respectively) and simple descriptive statistics were 
computed from the questions that used Likert scales. The resulting data were converted 
to tables and figures. Responses to the non-scaled questions were reviewed for any 


significant differences or similarities among the respondents. 


Like the responses to the non-scaled questions of the questionnaire, responses to 
interviews questions (Appendix D) were examined for trends indicating problems or 


frustrations experienced across participants. 
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B. FINDINGS 


1. General Web Survey 


The following pages present a detail statistical and graphical breakdown of 
responses to each question of the questionnaire. The first five questions were used to 
collected a profile of the squadron user, while questions six through twenty-one focused 


on computer usage. 


Sixty-four percent of the aviators tested in this survey indicated they spend 
between eight to ten hours at work per day. In addition the same percent indicated they 
are between twenty-five and twenty-seven years of age. Gender was not equally 
represented with males making up seventy-eight of the participants, however this is a 
good representation of a typical squadron with respect to gender. All participants 
indicated they are in the United States Navy with forty-two percent of them indicating a 
rank of Lieutenant (03) twenty-one percent as Lieutenant Junior Grade (O2) and twenty- 
one percent indicating a rank of Lieutenant Commander (04). See Figures 4.1 through 
4.5 for a detailed statistical and graphical breakdown of responses to each question of the 


questionnaire. 


a 
® 
a 
ra 
=) 
roe 
a 
a 
h 
= 
5 
“ 
a 
2 
E 
3 
z 


8-10 hrs 10-12 hrs >12hrs 


Figure 4.1. Hours worked 
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Question 1.1: How many hours per day, on average do you spend at work? 











Question 1.2: What is your age? 


Number of responses 





Figure 4.2. Age 


Question 1.3: What is your sex? 


Number of responses 





Figure 4.3. Sex 


Question 1.4: What is your branch of service? 


Number of responses 


Coast Guard Marine Corps 





Figure 4.4. Branch of Service 
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Question 1.5: What is your pay grade? 
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Figure 4.5, Pay grade 


All of the respondents indicated using a computer at home, with fifty percent 
doing so at work and thirty percent using a mobile computer. All reported using personal 
computers (PCs), twenty-one percent use a Macintosh, and twenty-eight use a notebook 
computer. Seventy percent use Windows 95/98™, with fifty percent indicating then use 
Microsoft Office 95/97. Browsers use was split evenly with fifty percent using either 
Microsoft Internet Explorer or Netscape. Obviously some of the respondents use two 
different browsers, but obtaining preference was not the goal of this question. See 
Figures 4.6 and 4.7 for a graphical breakdown of responses to each question of the 


questionnaire. 


Question 1.6: [use a computer at? Question 1.7: I use computer for what? 
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Figure 4.6. Computer use Figure 4.7. Computer Application 
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Activities perform on computer include: E-mail and Web access or Web browsing 
being performed by all respondents; word processing being used by seventy-one percent 
of the participants. Other applications include, forty-two percent using presentation 
graphics and twenty-one percent using financial programs and twenty-one percent 
indicating they play games. A small percent indicated that they used the computer to 


perform spreadsheet or DoD applications. 


Question 1.8: What type of activities do you perform on computers? 
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Figure 4.8. Computer Activities 


Sixty-four percent of the respondents indicated that they had never completed a 
formal computer course. Over half of the respondents, fifty-seven percent rated their 


computer skills as intermediate, while twenty-eight percent indicated themselves as 


novice. Question 1.9: Have you ever completed a formal computer course? 


Number of responses 





Figure 4.9, Formal course 
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Question 1.10: How would you rate your computer skills? 
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Figure 4.10. Computer skills 


Time spend on the Internet was distributed between zero to thirty, thirty to 
sixty minutes and one to three hours, each of these categories received fifty percent, 
thirty-five percent and fifteen percent respectively. Sixty-four percent indicated they 
browse or search the Web several times daily and thirty-six percent classified themselves 
at once daily. Seventy-one percent spend two to ten minutes on a Web site once they 
reach it, twenty-one percent spend zero to two minutes, and eight percent spend ten to 


thirty minutes at a site once they get there. 


Question 1.11: In total, approximately how much time to you spend on the internet? 


Number of responses 


0-30 min 30-60 min 





Figure 4.11. Internet usage 
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Question 1.12: Approximately how often do you Browse or Search the Web? 
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Figure 4.12. Web access time 


Question 1.13: When you get on a Web site, how long do you stay there? 


Number of responses 


2-10 min 10-30 min > 30 min 





Figure 4.13. Web site visit time 


Table 4.1 shows the ranking for actions taken once the respondents are finished 
looking at a Web site. Seven out of fourteen respondents indicated that then jump to 
another Web site without using a link as the most preferred action. The second most 


preferred action indicated by the respondents is to follow a link to another Web site. 
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Question 1.14. When you are finished looking at a web site, do you (rank in 
order, with 1 being most used action): 


Responses 


eEREP RRR PT TT | 


Follow a query-retrieved| 2/3) 4]4/2) 3 12 He 3/4/2)41/3 
suggestion 


Jump to another Web |3/1/3] 1/1] 4 3/3/14 
site without using a link 
Follow a link to another 113/3]2 21/2;4/2 
Web site 
[Giose the browser a]4[e|2|4ja|ali[afa[a]i[st4[ 250 | 


Table 4.1 Most Popular Action Taken When Finished Viewing a Web —— 



















Each user was asked to rank order common web page problems that detract from 


the page’s usability the most. Table 4.2 shows the overall results to this question. 


Question 1.15. Regarding the graphic user interface used on Web sites you 
visit, what do you find most detracting? (rank in order, with 1 being most used 
action): 


Responses 
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Table 4.2 Regarding GUI used on Web Site — Most Detracting 







This data confirms Nielsen’s points about Web site design. It also verifies that aviation 


personnel want many of the same features as the users in Nielsen’s usability test. 
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Complexity of a Web site’s address was not viewed as a discouraging factor in 


whether or not participants visited or used that (Figure 4.14 and Table 4.3). 


Question 1.16: The complexity of a Web site’s address discourages me from 
Visiting/using that site.? 
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Figure 4.14. Web site address complexity 


Table 4.3 Response Statistics for Question 1.16 


Mean 


Median 
Std Deviation 





The next three questions explored how participants view computers with respect 
to the workplace. Twelve of the fourteen respondents indicated that computers do not 
make them more productive (Figure 4.15 and Table 4.4). Half of the participants 
indicated that computer increase their workload (Figure 4.16 and Table 4.5). Eight of the 
fourteen participants disagree strongly that computer reduce paperwork (Figure 4.17 and 


Table 4.6). 
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Number of responses 


Number of responses 








Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 





Figure 4.15. Productivity 


Table 4.4 Response Statistics for Question 1.17 


Mean 
Median 
Std Deviation 


Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 





Figure 4.16. Workload 


Table 4.5 Response Statistics for Question 1.18 


Mean 
Median 
Std Deviation 
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Question 1.19: Computers reduce paperwork? 


i) 
@Q 
wa 
J 
° 
a 
o 
- 
- 
OF 
he 
®@ & 
Q 
E 
= 
=z 


Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 





Figure 4.17, Computers Reduce Paperwork 


Table 4.6 Response Statistics for Question 1.19 


Mean 


Median 
Std Deviation 





General questions were asked in which the participants could write in responses 
(Appendix E). The first of these was to list three Web sites they like and give the reasons 
why. Eleven of the fourteen responses involved news related sites like ABC, CNBC, 
CNN, the Weather Channel, the Washington Post and ZDNet, while six participants 
specifically listed ESPN. The reoccurring comment for ESPN site was that it contained 
good links to many other sites of interest. There were two references to search engines, 
due to the weath of informaiton possible there. Other sites that were not preferred were 
shunned because of broken links, promise but no substance, gratuitious graphics, most 
news sites for not being interactive, and search engines for being low yield. Other sites 


were singled out for requiring too much time to find the desired information (i.e. non- 
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intuitive sites), non-current information, too much advertising, and sites that were not 
connected or hyperlinked. 

2. ACSA Specific Survey 

In the first section of this survey, participants were asked to respond to a set of 
statements specifically regarding the ACSA Web site. The first series of questions was to 
get an overall impression about the ACSA Web site. The following pages present a 
detailed statistical breakdown of responses to each question of the questionnaire. The 
first seven questions focused on information content of the ACSA Web site. These seven 
questions were used to explore participants’ perception of whether the ACSA Web site 
would the needs of the user. Most participants indicated that the ACSA Web site 


provided information that was relevant, accurate, complete, current, required, intuitive 


and easy to read. Figure 4.18 through Figure 4.24 and Table 4.8 through Table 4.14 show 


the statistical breakdown. 


Question 2.1: Information on the site is relevant? 
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Figure 4.18. Relevant Information 


Table 4.7 Response Statistics for Question 2.1 


Mean 


Median 
Std Deviation 











Question 2.2: Information on the site is accurate? 
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Figure 4.19. Accurate Information 


Table 4.8 Response Statistics for Question 2.2 


Mean 
Median 
Std Deviation 
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Figure 4.20. Complete Information 
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Table 4.9 Response Statistics for Question 2.3 


Mean 


Median 
Std Deviation 





Question 2.4: Information on the site is current? 


Number of responses 


Disagree 
Strongly Disagree 
nor Agree 





Figure 4.21. Current Information 


Table 4.10 Response Statistics for Question 2.4 


Mean 
Median 
Std Deviation 
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Figure 4.22. Information Meets Needs 
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Table 4.11. Response Statistics for Question 2.5 


Median 
Std Deviation 





Question 2.6: Information on the site is intuitive? 


Number of responses 


Disagree Neither 
Strongly Disagree Strongly 
nor Agree 





Figure 4.23. Information Intuitive 
Table 4.12. Response Statistics for Question 2.6 
Mean 


Median 
Std Deviation 





Question 2.7: Information on the site is easy to read? 


Number of responses 


Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 





Figure 4.24. Information Easy to Read 
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Table 4.13 Response Statistics for Question 2.7 


Mean 
Median 
Std Deviation 





Questions concerning general appearance of the ACSA Web site indicated general 
satisfaction. Site address was not considered too complex (Figure 2.25 and Table 4.15). 
Most agreed that scrolling was not too excessive (Figure 2.26 and Table 4.16). Link 
colors and colors used throughout the Web site were acceptable (Figure 2.27 and Figure 
2.28, Table 4.17 and Table 4.18). The amount of graphics used received mixed opinions 
but generally the Web site appears to have a good blend (Figure 2.29 and Table 2.19), 
however the user indicated that they generally did not want additional graphics added 
(Figure 2.30 and Table 2.20). 


Question 2.8: The site address is too complex? 


Number of responses 


Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 





Figure 4.25. Address Complexity 


Table 4.14 Response Statistics for Question 2.8 


Mean 


Median 
Std Deviation 











Question 2.9: The amount of page scrolling is acceptable? 
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Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 





Figure 4.26. Page Scrolling 


Table 4.15 Response Statistics for Question 2.9 


Median 
Std Deviation 


Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 


Figure 4.27. Link Colors 


Table 4.16 Response Statistics for Question 2.10 


Median 
Std Deviation 











Question 2.11: The colors used throughout the site are acceptable? 
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Figure 4.28. Site Colors 


Table 4.17 Response Statistics for Question 2.11 


Median 
Std Deviation 


Disagree Neither 
Strongly Disagree Strongly 
nor Agree 





Figure 4.29. Graphics 


Table 4.18 Response Statistics for Question 2.12 


Mean 
Median 
Std Deviation 
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Question 2.13: I would like to see more graphics. 
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Figure 4.30. More Graphics 


Table 4.19 Response Statistics for Question 2.13 


Mean 
Median 
Std Deviation 





The next two question concerned down load times between HTML and Java 
applets. Download times for the HTML format are generally acceptable (Figure 4.31 and 
Table 4.20), however, Java applet download times were viewed as unacceptable (Figure 


4.32 and Table 4.21). 


Question 2.14: Download times are acceptable (HTML format). 


Number of responses 


Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 





Figure 4.31. HTML Download Times 
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Table 4.20 Response Statistics for Question 2.14 


Median 
Std Deviation 





Question 2.15: Download times are acceptable (Survey Applet). 


Number of responses 


Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 





Figure 4.32. Survey Applet Download Times 


Table 4.21 Response Statistics for Question 2.15 


Mean 
Median 
Std Deviation 





Question concerning navigation through the ACSA Web site received less 
favorable response. Navigation support was considered to be lacking in that there were 
no links to questions or common problems (Figure 4.33 and Table 4.22). Navigation 


from one web page to another was considered clear (Figure 4.34 and Table 4.23). 
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Question 2.16: Navigation support is acceptable. 


7) 
o 
a 
r= 
° 
a 
a 
o 
ho 

= 
° 
- 
® 

Q 
E 
3 

P4 


Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 


Figure 4.33. Navigation Support 


Table 4.22 Response Statistics for Question 2.16 


Mean 
Median 
Std Deviation 


Number of responses 


Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 


Figure 4.34. Navigation Intuitive 


Table 4.23 Response Statistics for Question 2.17 


Mean 


Median 
Std Deviation 




















The next series of questions centered around individual Web page design and overall 
presentation. These questions focused attention on the seven pages which comprise the 
ACSA Web site. Web site design and overall presentation of for each page was rated and 
generally received favorable marks. The Main page design and overall presentation 
received positive marks (Figure 4.35 and Table 4.24). The HTML Survey page 
receieved a very good rating with ten of the eighteen respondents rating it at very good ( 


Figure 4.36 and Table 4.25). 


Question 2.18: Main page design/overall presentation. 
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Figure 4.35. Main Page Design 


Table 4.24 Response Statistics for Question 2.18 


Mean 
Median 
Std Deviation 
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Question 2.19: HTML survey page design/overall presentation. 
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Figure 4.36. HTML Survey Page Design 


Table 4.25 Response Statistics for Question 2.19 


Mean 
Median 
Std Deviation 





The Survey Applet page design received mixed rating with five of the eighteen 
participants indicating it was a poor design. Primary reason for such a low rating was 
primary due to flashing text, undulating text, and scrolling pages (Figure 4.37 and Table 
4.26). Several participants indicated that downloading the Survey Applet was excess and 


often timed out due to server none response errors. 


Question 2.20: Survey Applet page design/overall presentation. 
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Figure 4.37. Survey Applet Page Design 
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Table 4.26 Response Statistics for Question 2.20 


Mean 
Median 
Std Deviation 





Participants’s responses to design of the Results Applet page indicated gereral 
satisfaction, however, eight participants left this question blank due to non-access to the 
page. Several questions did arise during the interview which question why squadron 
personnel would have access to the survey data. This page was designed to give 
Commanding Officer the ability to view results of a question and make comparisons with 
other units. Due to the limited availablity of Commanding Officers to participant in the 
ACSA usability survey it was included soley in a effort to evaluate it as a useful tool in 


the future. Ten of the eighteen responses are presented in Figure 4.38 and Table 4.27. 


Question 2.21: Results Applet page design/overall presentation. 
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Figure 4.38. Results Applet Page Design 


54 








Table 4.27 Response Statistics for Question 2.21 


Mean 
Median 
Std Deviation 





Participants’ responses to questions concerning the design/overall presentation of 
the Help page was acceptable (Figure 4.39 and Table 4.28). Design and overall 
presentation with respect to Secure HTML Survey page received mixed results due to a 
lack of understanding between the HTML Survey page and Secure HTML Survey page 
(Figure 4.40 and Table 4.29). Participants’ response to design and overall presentation of 
the ACSA newsgroup page indicated a lack of understanding of what the page was design 
for (Figure 4.41 and Table 4.30). Several responses to these two questions were left 


blank and only eleven participants indicated a response. 


Question 2.22: Help page design/overall presentation. 


Number of responses 


Poor Acceptabie Good VeryGood Excellent 





Figure 4.39. Help Page Design 


Table 4.28 Response Statistics for Question 2.22 


Mean 


Median 
Std Deviation 











Question 2.23: Secure HTML Survey design/overall presentation. 
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Figure 4.40. Secure HTML Survey Page Design 


Table 4.29 Response Statistics for Question 2.23 


Mean 
Median 
Std Deviation 


Number of responses 


Acceptable Very Good = Excellent 





Figure 4.41. ACSA Newsgroup Page Design 


Table 4.30 Response Statistics for Question 2.24 


Mean 


Median 
Std Deviation 








The next series of questions were to have each participant rate each page 
according to functionality and ease of use. Instructions of Main page received only minor 
favorable results due too the technical tone of instructions and lack of any statements 


concern confidentiality (Figure 4.42 and Table 4.31). 


Question 2.25: Functionality & Ease of use for Instructions on Main page. 


Number of responses 


Poor Acceptable Good VeryGood Excellent 





Figure 4.42. Main Page Functionality 


Table 4.31 Response Statistics for Question 2.25 


Mean 


Median 
Std Deviation 





The next two questions compared two different survey formats, HTML and Java 
Applet. Participants indicated a preference for the HTML survey for several reasons, 
such as, easy of reading, download time, color separation for each question and not 


having too scroll to read each question (Figure 4.43 and Table 4.32). 


oT 











Question 2.26: Functionality & Ease of use for HTML Survey page. 


Number of responses 


Poor Acceptable Good Very Good Excellent 





Figure 4.43. HTML Survey Functionality 
Table 4.32 Response Statistics for Question 2.26 
Mean 


Median 
Std Deviation 





A’ majority of the participants indicated that the Survey Applet format was 
acceptable but not as good as the HTML format. Responses during the interview 
indicated that due to fashing text, alignment while reading and selecting the answer, 
layout of page and scrolling to read each question contributed to negative views (Figure 


4.44 and Table 4.33). 
Question 2.27: Functionality & Ease of use for Survey Applet page. 
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Poor Acceptable Good VeryGood Excellent 





Figure 4.44. Survey Applet Functionality 
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Table 4.33 Response Statistics for Question 2.27 


Mean 
Median 
Std Deviation 





Participants’s responses to functionality and ease of use of the Results Applet 
page indicated gereral satisfaction, however, eight participants left this question blank due 
to non-access to the page. Many participants indicated during the interview that the 
Results Applet was an impressive tool for Commanding Offiers to use. Several 
participants made postive comments on the ability to view data in many different queries. 


Ten of the eighteen responses are presented in Figure 4.45 and Table 4.34. 


Question 2.28: Functionality & Ease of use for Results Applet page. 


Number of responses 


Poor Acceptable Good VeryGood Excellent 





Figure 4.45. Results Applet Functionality 


Table 4.34 Response Statistics for Question 2.28 


Mean 
Median 
Std Deviation 
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The Help page was not hard to understand and received favorable marks, 
however, participants indicated a need for a table of contents and links to major points in 
the help page. Several participants challenged the naming of Help page while providing 


nothing more than a frequently asked questions page (Figure 4.46 and Table 4.35). 


Question 2.29 Functionality & Ease of use for Help page. 


Number of responses 


Poor Acceptable Good VeryGood Excellent 





Figure 4.46. Help Functionality 


Table 4.35 Response Statistics for Question 2.29 


Mean 
Median 
Std Deviation 





Functionality and ease of use with respect to Secure HTML Survey page once 
again received mixed results due to a lack of understanding between the HTML Survey 
page and Secure HTML Survey page (Figure 4.47 and Table 4.36). Participants’ response 
to functionality and ease of use to the ACSA newsgroup page indicated a lack of: 
understanding of what the page was design for (Figure 4.48 and Table 4.37). Several 
responses to these two questions were left blank and only eleven participants indicated a 


response. 
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Question 2.30: Functionality & Ease of use for Secure HTML page. 


Number of responses 


Poor Acceptable Good VeryGood _—_ Excellent 





Figure 4.47. Secure HTML Functionality 


Table 4.36 Response Statistics for Question 2.30 


Number of responses 


Poor Acceptabie Good VeryGood Excellent 





Figure 4.48. ACSA Newsgroup Functionality 


Table 4.37 Response Statistics for Question 2.31 


Mean 
Median 
Std Deviation 
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Two questions were used for comparison between the HTML survey and Survey 
Applet. The HTML format was perferred over the Survey Applet (Figure 4.49 and Table 
4.38). The Survey Applet was less desired by most the participants due to flashing text, 
alignment while reading and selecting the answer, layout of page and scrolling to read 


each question contributed to negative views (Figure 4.50 and Table 4.39). 


Question 2.32: HTML is easier to use. 


Number of responses 


Poor Acceptabie Good Very Good = Excellent 





Figure 4.49. HTML Comparison 
Table 4.38 Response Statistics for Question 2.32 


Mean 
Median 
Std Deviation 





Question 2.33: Applet easier to use. 
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Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 





Figure 4.50. Applet Comparison 
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Table 4.39 Response Statistics for Question 2.33 


Mean 
Median 
Std Deviation 





The last question asked participants to rate how satisfied they were with the 
ACSA Web site (Figure 4.51 and Table 4.40). Fifteen of the eighteen responses were 
either satisfied or very satisfied. From interview comments the Web site was viewed to 
be innovative and move in the right direction for collectiong survey data. Several Safety 
Officers commented that the Web site would be welcomed in their squadrons and looked 


forward to having it as tool to produce mini-surveys in the future. 


Question 2.34: Overall, how satisfied are you with the ACSA Web site? 


Number of responses 


Disagree Neither Agree 
Strongly Disagree Strongly 
nor Agree 





Figure 4.51. Satisfaction of ACSA Web Site 


Table 4.40 Response Statistics for Question 2.34 


Mean 
Median 
Std Deviation 
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V. CONCLUSION AND RECOMMENDATIONS 


A. CONCLUSION 

Knowledge of usability of Web based information tools by aviation personnel 
holds value for designers of Web pages and sites, as well as the user of those applications. 
By knowing what to expect and demand, users can shape the future of these designs, and 
by knowing what users want and desire in a Web site based application, designers can 
provide systems that are usable and effective. Several conclusions can be drawn from 
survey results and interviews. 

e Simplicity is essential. Users preferred the simple HTML format to the 

complex Applet format. 

e Content is important. If content is lacking the user will look elsewhere. 

e Security. Confidentially and anonymity must be intuitive. 

e Functionality. Each navigation button must be understandable. 

e Download time. Must understand end-user system requirements. 

e Page layout. Excessive scrolling of pages must be avoided. 

e Usability is the key to success. Proper usability evaluation will save on 

development and redesign cost. 

B. RECOMMENDATIONS 

The information obtained from the literature and the insight gained from the 
surveys and observations lead to several suggestions. These concluding statements 
should provide the stimulus for change supported by the collated data. The results of this 


study and the conclusions drawn from the suggest that the actions listed below will 
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enhance the ACSA Web site and make it a much easier application for collecting survey 
data. 

e Eliminate the Survey Applet from the ACSA Web site. This will decrease 
loading times and requirements for Java plug-in modules. 

e Rewrite the introduction page to include a statement concerning survey goals 
and objectives, confidentiality of responses, and anonymity of individuals. 

e Redesign the help page to include a table of contents, links and list of 
acronyms. 

e An introduction page needs to be included before the HTML survey and 
Survey Applet. The page should highlight survey objective, safety goals and 
requirement for data. 

e Removes help links from HTML survey and Survey Applet. These links only 
send you to the Help page which do not address any survey issues. 

e Make either HTML survey or Secure HTML survey available, not both. 
Confusion exists between the unsecured and secure formats. 


e Provide links to other related Aviation Web site. 


C. FUTURE RESEARCH 

The recommendations offered are an indication that future work can be done on 
the ACSA Web site. Research into the saving a partially completed survey is probably 
necessary in the event an individual cannot complete the survey and must return. Another 
area of research applicable to the ACSA Web site concerns the effectiveness and 


efficiency of administering the ACSA survey electronic over a traditional paper. 
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A study relating the effects of linked information should be performed. This study 
should attempt to determine whether users prefer links to information that may not be 
entirely accurate, or no link at all. It could also explore the effects of linked information 
on the referring Web site. 

The lack of literature in this area suggests a fertile area to be probed by future 
researchers. Additional bodies of data, large enough to be statistically significant should 
be gathered and analyzed. Continued surveying with the current surveys, or a similar 
updated survey of this type, would provide further insight into the aviation community as 


a user of this system. 
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APPENDIX A. GENERAL WEB SURVEY 


Note: Due to formatting differences this survey changed slightly in appearance. 
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GENERAL WEB SURVEY 


Please read the following before continuing........... 


Information gained by this survey will be used in a thesis aimed at creating 
a better Web interface design for the Aviation Command Safety 
Assessment (ACSA). Accurate information is essential to developing an 
accurate profile of indented users. Feedback will also be integrated into 
the ACSA Web site. The information obtained will also allow us to 
profile the typical users squadron level. The information obtained in this 
survey is and will remain anonymous; no names will be taken or 


associated in any way to the information gained. 


This survey should take approximately 10 to 15 minutes to complete. 


Thank you for taking the time to participate. 
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What is your age? 


Under 25.......... CL] 25 to 27... eee 0 28 to 30.......... Cc 31 to 33.......... | 33 or older.......... C] 


What is your sex? 


Male......... im Female.......... O 


What is your branch of service? 


Army........- | Coast Guard.......... a Marine Corps.......... C Navy......0000 CJ Other......... 


EV 2 Geen Creer oe mem er CO a Macintosh................00-5 Oo a Notebook..........:00.0c008 CT] 
a Palm Top........s0--+L] Windows 95/98......0.....04+ im Netscape Browser............] 
MS Internet Explorer....[-] MS Office 95/97...........4. C] 


Other (please write in): 


What type of activities do you perform on computers (please check all that apply, whether at work or at home): 


Eemail......0...cecceeee sees CT] Word Processing.............6.00 C Web Access/BrowSing............s000006 CT 
Gaines echenledscuensonialed Presentations Graphics............[] DoD Applications (Supply, Travel).....[[] 
Spread sheet........... oe Financial Programs..............4. CJ Other niceties eae eee C] 








Please take a few minutes to complete this survey. Your specific answers will be completely anonymous, but your views, in 
combination with those of others, are extremely important. 





9. Have you ever completed a formal computer course? 


Yes....c...0 LC] Nou... .e00e oO 
10. How would you rate your computer skills? 

Novice.......... C] Intermediate.......... C] Expert.......... CT] 
Internet use: 


11. In total, approximately how much time per day, on average, do you spend on the Internet (i.e. Browsing or searching the 
web) for any purpose? 


0-30 minutes.......... | 30-60 minutes......... CL] 1-3 hours.......... CJ More than 3 hours......... oO 


12. Approximately how often do you Browse or Search the Web? 


Several times daily................00s000 oO Once daily......... C Less than five times per week.......... im 
Less than ten times per month......... CO Nevel.......c0.0000e im 


13. When you get to a Web site, how long do you stay there? 


0-2 minutes.......... im 2-10 minutes......... CO 10-30 minutes.......... CJ More than 30 minutes......... Cc] 


14. When you are finished looking at a Web site, do you (rank in order, with 1 being most used action): 


Follow a query-retrieved suggestion (i.e go back to a search engine and follow a link from there)................... CJ 
Jump to another Web site without using a link. ............ cece ce ccee ee er eee nce reece eeeteereeesesereeecesenesseaesenenenenee [J 
Follow a link to another Web sites. .6.00ccciccecee sabe sees cdocuaceweldacepeega ad eviesd ones Javsecrcdauases¥euevedsevnsesdsostest C] 
Close the: DIO WSEDsgiiev crete ansdgec Sek ds Reh edule wed Over eed seca ded i dLs di ulatas becdeas Skee ahha outdone ese LE] 
N/A, I do no access the WeD......c.cccccecsccsccssccssecsecessesssessvssessecaseversesssesecaseecerseraceesserecesseasarsacees [] 


15. Regarding the graphic user interface used on Web sites you visit, what do you find most detracting? 
(Rank in order, with 1 being most detracting) 


Long download times.............--..000008 C] Counter intuitive links......... i Non-standard link colors.......... C] 
Long scrolling pages.............0:000000e C] Lack of navigation support...[[] Poor color schemes............+++ CT 
Scrolling text, marquees, ads, constantly running animation...................0.0665 | 











Disagree Neither Disagree Agree 
Strongly nor Agree Strongly 
16. I view Web site address as complex and hard to USe ...........sssceseesereees Lnksiatnice Dies shiteteans Dicseieviestaad AS eceat anaes 5 
17. Computers make me MOTE PrOdUCtiVE ...... elec reeceeceseeeeeteteteereenteeeres Vessativecaaete se Qessseitcaeedess Siriaas Cee 5 
18. Computer increase my Workload 00... sssecessecesessessnesesescasscnatsecesseeees L ererreoreres 2D sachussdeisveds Sisideaneae: 4a cscteengesees 5 
19. Computer reduce paperwork 0... csssccesessercecresecscsenecssceonsenesceresereseees Tite sechecdisis Qiidietineriacss Bikes Aes eceacsetises 5 


20. List 3 Web sites you like and give reasons(s) you like each one (you do not have to use the exact address) 


NOTE: If you can not think of 3 sites, please list 3 features you like about Web sites. 


21. List 3 Web sites you do NOT like and give the reason(s) for each one (you do not have to use the exact address) 


NOTE: If you can not think of 3 sites, please list 3 features you do NOT like about Web sites. 
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APPENDIX B. LOG ON PROCEDURES 
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Welcome to the Aviation Command Safety Assessment (ACSA) 


We would like to thank you for participating in this usability analysis. Our goal is to learn how we can improve this web site 
for the fleet. Your comments will assist in making the ACSA another valuable resource for naval aviation. The survey 
consists of nine biographical questions and 61 survey questions. We assume you are familiar with basic browser/web site 
navigation. Due to the length of the survey, this analysis/interview should take roughly 30-45 minutes. 


I. Completing the Usability Analysis HTML Format 


1. Open your browser of choice to begin, either Netscape or Internet Explorer. 
2. Enter the following Internet address hitp://spitfire.avsafety.nps.navy.mil to access the ACSA home page. 


3. After accessing the ACSA home page, you will be asked to enter a User Name and Password. Enter the following: 


Use name: Don’t 
Password: XXXX 


The ACSA home page provides general information for the user. Please take a moment to read this page before proceeding. 
4, Select HTML Survey to begin. 


5. You should now be viewing a survey form containing Biographical Information and Survey Questionnaire sections. 


For purposes of this evaluation answer the Biographical Information questions with your current personal data. 
6. Answer all Survey Questions from the perspective of your last flying command. Use the drop-down list-boxes to 


select your appropriate entry for each question. 


II. Submitting your Survey 


1. Click the SUBMIT button at the bottom of the page once. This will open a dialog box requesting your squadron 
number. 


Note: If you made an error completing the form, or forgot to answer a question then the program will alert you to complete 
the appropriate question. After completing this question re-click the SUMBIT button 


2. Type your squadron number for the squadron number and then click OK. This will open another identical dialog box 
requesting your survey number. Your survey number is 


3. Type your assigned survey number and click OK. If your submission entries were correct, you will see the Thank You 
page, or else you will see a warning page. 
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III. Viewing Results 

The ACSA program provides commanders in the field the opportunity to query the ACSA database. The squadron ASO and 
CO will have a special password to access this page. 

1. Select Results Applet to view results. You will get a warning that Authorization failed. Enter the following: 


User Name: Commander 
Password: XXXXXXXX 


2. To view results: 
1. Select a question from the drop-down box. 
2. Click SUBMIT. 


Queries can be limited by selecting additional query parameters from the three remaining drop-down boxes. Note the first 


query takes several (10 — 15) seconds to process over the network, but subsequent queries are much faster. 


Thank you for taking the time to complete this survey. 


We are interested in your comments! 
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APPENDIX C. 


AVIATION COMMAND SAFETY ASSESSMENT (ACSA) USABILITY SURVEY 


Note: Due to formatting differences this survey changed slightly in appearance. 











ACSA USABILITY SURVEY — 


Please read the following before continuing........... 


Information gained by this survey will be used in a thesis aimed at creating a better Web 
interface design for the Aviation Command Safety Assessment (ACSA). Accurate information 
is essential to developing an accurate profile of indented users. Feedback will also be integrated 
into the ACSA Web site. The information obtained will also allow us to profile the typical users 
squadron level. The information obtained in this survey is and will remain anonymous; no 


names will be taken or associated in any way to the information gained. 


This survey should take approximately 20 to 30 minutes to complete. 


Thank you for taking the time to participate 
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Regarding the ACSA Web site, please respond to each of the following: 


Disagree Neither Disagree Agree 
Strongly nor Agree Strongly 
1. Information on the site is relevant...............eecsceeseetesseececeeeeeeeeeseeeteeees Teeaten DB Sisi tsi og Binsin ees ee 5 
2. Information on the site is aCCUTate ..........seescsssceceettesesenescestesseesereesers | Beerrrere D sinscesteesss Dvieossaces sete zee pre 5 
3. Information on the site is complete... eeeseessesceeeseessenseeeeserseeees | eee reer Disccbeeveess; Stites’ Ay istesteluexs 5 
4. Information on the site is CUITENE ........sscsseeessecesesetenereeeenetetetateceseseees Woes D veatVissotetsss Diener A eisicecectecey 5 
5. Information on the site meets my needs...............:sseccsceeseeseseeerseseeeeaees | ere erent eee Bi cielalasies 4 oecaceine: 5 
6. Information on the site is intuitive 0.0... eeseeeteeeeeteeessenseeterseeaees | oopeeerterepes Dir essedeecuuats DB inssseseevsts Cena r rare 5 
7. Information on the site is easily readable... ees seeseeeseeesesereeeseeeees Tigi ietes Qescsterbennied Sistiheases Bis i08 Stas: 5 
8. The site address is too COMPLEX... eee seeseeessesssceeteseeseseeeseseeseseeeneaes  eereepr error iD Seisvesiasecss Savistereustestg We one beseees 5 
9. The amount of page scrolling is acceptable .0..........sscsccssseesseeeseeessseees Togeasdies Deas sents iB saersencaaes Geiss eetuased 5 
10. The link colors are acceptable... eee secereeeececeneecesneeeeenscatennenetets | neers Disdcccesstesaty Bj checcsaphesly Bie ae 5 
11. The colors used throughout the site are acceptable... eee eseeeees | eer reaaes Disscteccs ieee B ecessuebiosened Ai scstateincety 5 
12. The amount of graphics is acceptable ........ te eseseecesesetereneteteseneeeees Weaiieienees Disienidaieleraas Siewssisdetives Ao seBeactecets 5 
13. I would like to see more graphics... eessseseeeseseeceeeeescteeesesnsaceneesens | ere i” heneea 2 haceienvien: Bresseivveetrerts A iiimssceits 5 
14. Download times are acceptable (HTML format) ..............ecscsseseereeees Loaves Draieselen Bi Wwestsiceeste ca nee ae 5 
15. Download times are acceptable (Java applet) 0.0... sssssesereceeeeeeeeeee Wess iis eetes Digseasevciavines 3B eissecetenests A cectizessdvines 5 
16. Navigation support is acceptable: 0.0... eeecesceseceseseneecesesssesesesseseseetenes | eer eer Questa tesesees Sescedseinite hee Bed. idicssies 5 
17. Navieation through the Site 18 INtUItIVE oc sescssesetsrererseeeneneeeeeenee Liisciies Qirasiesasieiees Shadi Alok ane 5 
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Please take a few minutes to complete this survey. Your specific answers will be completely anonymous, but your views, in 
combination with those of others, are extremely important. 





Please answer the next set of questions regarding how you feel about each of the following specific matters? 


Go to the ACSA Web site located at http://spitfire.avsafety.nps.navy.mil 


Web site design/Overall presentation (Please circle one number for each statement) 





Poor Acceptable Good Very Good Excellent 


182, Main page. Ac....008 Ses ee tenons ee eo a eee sane nee Diosesebecntict cee aeneers Ae Sate, 5 
19. “HYME: Stitvey: page: 2.-./0se cise, slender tise cintnnnie baddest | Peeeerrraey De dacuate s Sena AKANE 5 
20: Survey Applet page 24. :0.:.:ba2si endo herd ee SI A ed ae Vie Doligiverles Bonn ees 4 ote 5 
21 Results Applet pace x:2:cccstseeetveri id Soest ASA | Re eereReere Db ieeveets Biante LS eer 5 
22) Help Pace avinissasiivesuraats ih cobesesehoascsedsndeivaresdscdedaraesbiteonsbsaebereside stbneesdatevtosnasbseves Dsasesdesdetts 2 dieu B sitestestieves Anti ds 5 
23. Secure HTML Survey page.........sccccecsserssccssssserseenncsessscsesssssnessssnsreenseesseees | eee Disceteseenatiess 3 ieeteniies See rere 5 
24. ACSA NewS gTOup Page ic. ccscsscscdcstessestracessoncseasssbevedacenvssvarsassarsvenstcedbvesnecestvers L esietieecdetes D  sscctiiastnetes B eetetetesdouts A iiaesscatstneies 5 


Functionality/Ease of use (Please circle one number for each statement) 


Poor Acceptable Good Very Good Excellent 


25. Instructions On Main page ...........ccessscssessssecsseserssecssesesseesssseseessesseceeseeseeeeseees La tieveneeys Ds cesesestesites Dabstenlocenstes CEE ornOre 5 
26. THEME Survey pages czciccssc.c.cc42pitcs Sissies ctessidaniecesdsannssvdstabe.desicaedvadtvanandensen’ Leek det Dissssecivevertss SD cecleteee ees Bis cates 5 
27. ‘Survey Applet page o.cscc.scccsasciieeces iusccueccts ccescensaeteseee Debedcsessegeaseaseessvactaewsasas | ener es Dis usexs: Bisccciv Acetate 5 
28. Results Applet page...........ccecssssssserscserssssesssesssecesesssecsesersetsssesenensereseeeseneass oD resesseesnesee Dsesdehesttves Dieses ¢, Sea eer 5 
29: HIP Page iisnstesideietinchas caves shlestsosbvavevenscovaclesdcrssiseesctscesusecsaruseitvsdstectedsdeietssvaves | Geen 2 ehSiisedesess SD iscudtesesves Fs asses ssestes 5 
30. Secure HTML Survey page......cccsessccscssesssssccesssesscnssssecessssensvensserseneesees | eee ee Lewin diies Sates terre 5 
31. ACSA NEWS atOup page ics. .icasccesoscstecvedcestss sucesdessunsnseiedescdisbesans desozadesessioatindes Dixssesdteiice Qivieiesdesnties Sisseiieniesds Cre reerera ee 5 
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Please take a few minutes to complete this survey. Your specific answers will be completely anonymous, but your views, in 
combination with those of others, are extremely important. 





In comparison to the HTML Survey and Survey Applet Disagree Neither Disagree Agree 
Strongly nor Agree Strongly 

32. HTML is easier to Use .........-.secs-csesesecsssseessesseccncaevososaeseesenescsssacnsessesessescasnece | Parereecrt ese D sistcaeriton Bisecedvcusabades Aes chetiiseuess 5 

33; “Applet Gasier to USC siisicdasssesazecsesscestvncseonentsecandegenvsnsilardevecoapheoshgenn sdveusdasvaseazays | eee 2 cestuisieaitass Sere cistsdeysese Be Socossietiess 5 


34. Overall, how satisfied are you with ACSA Web site? (Please circle one number) 


Very Neither Satisfied Very 
Dissatisfied Dissatisfied nor Dissatisfied Satisfied Satisfied 
1 2 3 4 5 
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APPENDIX D. INTERVIEW QUESTIONS AND SIGNIFICANT RESPONSES 


Question: Did you find the portion of the ACSA that you used to be an easy tool for 
completing the survey? 


Responses (HC-3, HC-11 and HSL-45): 


e Yes, survey (HTML) was easy to access and easy to complete. 

e Yes, very logical and something that will add value to the fleet. 

e Yes, great Web site. 

e Yes, but why are there two HTML formats? 

e Yes, quick to use and easy to submit. 

e Yes, I thought it was straight forward and intuitive. 

e Yes, super graphics and nice layout. 

e No, Java plug-in take to long to load and download times are excessive. 
e No, what is a ACSA newsgroup. 

e No, Survey Applet won’t run right, but HTML format is very nice. 


Responses (HC-2, VAW-120 and VAW-123) 


e Yes, way to go— look forward to it getting to the squadrons. 

e Yes, will make my job easier as Safety Officer. 

e Yes, excellent tool to have in my bag of tricks. 

e Yes, CO’s will fun it. 

e Yes, but is my answer’s secure. 

e Yes, but why the different formats (HTML, Secure HTML, Survey Applet)? 
e Yes, nice work on Web site and good graphics. 

e Yes, but what happens if there is a power outage while taking the survey? 
e No, Survey Applets take too long to load. 

e No, Help is very helpful. 

e No, too much scrolling — can’t reset my screen. 


e No, downloading plug-in is a pain in my %#*. 
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Question: Was there anything that you found particularly difficult to do or 
understand? 


Responses (HC-3, HC-11 and HSL-45): 


e Not really 

e No, easy to use survey 

e No, but I couldn’t get Survey Applet to work at first. 

e Yes, what is ASCA newsgroups. 

e Yes, difference between Secure HTML and HTML survey. 
e Yes, help is not a help function. 

e Yes, Results applet takes to long to download 


e Yes, why do I have navigation buttons to items I will not access in the future 
(i.e. Results applet, ACSA newsgroup)? 


e Yes, how is the data secured? 
e Yes, what protects me from someone finding out what my answers are. 


Responses (HC-2, VAW-120 and VAW-123) 


e No, but the Main page instructions are too technical. 

e No, but the help menu is not index to find anything. 

e Yes, what is ASCA newsgroup and who uses it. 

e Yes, Java plug-in would not load and finally took 30 minutes. 

e Yes, what protects me from someone knowing my responses? 

e Yes, why is there a Secure HTML survey is it better than HTML survey? 


Question: If you could change one thing about the ACSA Web site to make it easier 
to use, what would it be? 


Responses (HC-3, HC-11 and HSL-45): 


e Delete the Survey Applet, take too long to load. 
e Delete flashing text and waving text in Survey Applet. 
e Make the Help function better 
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e Expand the list of terms. 

e Too much scrolling to read information. 

e No, easy to use survey 

e Survey Applet does not have “select from the following”. 

e Add Likert scale at the top of each survey 

e Flashing titles are annoying 

e Options to go back and change selection “Need back button”. 
e Small type size and too scrolling. 


Responses (HC-2, VAW-120 and VA W-123) 


e Main page scrolling too fast to read. 

e Make statements that answers are will be confidential in open introduction. 
e State purpose of survey. 

e Applet page should have note to continue with “Tab Questions”. 

e Need Likert scale at top of page. 


e Main page — need statement about “your answers will remain confidential, no 
attempts will be made to identify you or your organization”. 


e Help need some type of index. 

e Result page — need to select both coast at once. 

e Delete loading Java plug-in, what a waste of time. 

e Either keep Secure HTML survey or HTML survey, but not both. 
e One need one type of survey - HTML works just fine. 

e Too much scrolling — make page that fix monitor. 


Question: Are there any features you would like to see added to the ACSA Web 
site? 


Responses (HC-3, HC-11 and HSL-45): 


e Links to other aviation safety issues. 
e Access to Results page by Safety Officer and ASO’s. 
e A better help feature not just a list of questions answered. 
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Responses (HC-2, VAW-120 and VA W-123) 
e Larger text size. 


e Looks great the way it is. 
e [like the lay out but scrolling is a problems. 
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